Multi-criteria Assessment of Ecological Process Models

The Pareto Optimal Model Assessment Cycle (POMAC), a multiple-criteria model assessment methodology, is described for exploring uncertainty in the relationships between ecological theory, model structure, and assessment data. Model performance is optimized to satisfy, simultaneously, each component of a vector of assessment criteria (model outputs), rather than the usual procedure of optimizing performance with respect to a single criterion. Pareto Optimality is used to define the vector optimization. The Pareto Optimal Set reveals which combinations of assessment criteria the model can satisfy simultaneously. Binary interval error measures, which classify whether a parameterization result is within an acceptable range of values, are defined for each criterion. Their use masks small differences in the performance of different parameterizations, allowing the Pareto Optimal Set to reveal conflicts in ability to achieve simultaneously different collections of criteria. POMAC improves the researcher’s ability to detect deficiencies and locate their sources. It is more stringent and informative than traditional model assessment procedures because it uses multiple criteria without weighting and aggregating them. The Pareto Optimal Set reveals the presence of deficiencies through the model’s inability to satisfy all the criteria simultaneously. POMAC then guides the researcher in locating deficiencies in: inadequate selection of component ecological hypotheses underlying the model, inadequate mathematical representations of these hypotheses, inadequate parameterization, poor selection and formulation of the assessment criteria, or combinations of these. In an example, POMAC is applied to the spatially explicit canopy competition model WHORL using ten assessment criteria. Each criterion was selected to provide information on different aspects of WHORL’s functioning: three stand height distribution criteria, three crown morphology criteria, and four criteria focusing on stand competition’s characteristic differentiation of growth rates. The Pareto Optimal Set was generated using simulated evolution optimization. POMAC revealed deficiencies in both the model structure and its assessment criteria, leading to an improved model and better understanding of its effective domain.